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Abstract 



Outcomes assessment is increasingly important to regional accrediting agencies, which are requiring self- 
study reports to not only analyze the outcomes assessment efforts but also use assessment results to 
improve programs. My presentation will illustrate the process in a university-wide required business 
communications course by discussing the problems of setting standards, enlisting faculty support, and 
establishing a consistent portfolio rating system. Attendees will receive handouts of the exit standards, 
rubrics, and assessment results. 



Presentation Description 

Outcomes assessment is increasingly important to regional accrediting agencies, which are requiring self- 
study reports to not only analyze the outcomes assessment efforts but also use assessment results to 
improve programs. Also, foundations and corporations who donate money to support business 
communications programs more frequently request evidence that their funds are well used and that, in 
fact, students are learning. 

At my university, we have a multi-course Communications Skills program that is required to develop 
multiple-measure assessments. One part of the assessment is to measure student achievement at the end of 
Communications Skills V, a university-wide business communications course that is also end of a five- 
course integrated skills sequence. In each course we assess student achievement in reading, writing, 
speaking, and listening. Past results show that student writing needs to improve, yet the average grade in 
the course is A-. We asked ourselves whether students are really meeting course standards to produce 
work that is “workplace ready.” Are our standards rigorous enough? Initially, we adopted a policy that 
each student must pass the writing component to pass the course, and that student writing, which is 
collected in a portfolio, will be assessed by second readers to see if students are meeting course exit 
standards. 

Several problems arose: First, the decision to use second readers made the faculty nervous (“you’re 
second-guessing me”). Second, we had to decide what and how much to assess in a course that requires 
group reports, PowerPoint presentations, letters, memos, and case studies. Third, we had to calibrate 
scoring to ensure consistent ratings. 

I will discuss the process we developed to address these problems and our tentative solutions. I will also 
share results of Spring 2002 portfolio assessment and how we are using them to improve the program. 
Finally, I will provide handouts of the exit standards and rubrics we used, and so that ABC members may 
adapt them to their home institutions. 

The Assessment and Results 

The purpose of the portfolio assessment rating session was to assess student outcomes to learn how 
effective our instruction is and to base program improvement on the results. On Saturday, April 20, 2002, 
12 full- and part-faculty who teach CO230 Communications Skills V performed a criterion-referenced 
assessment of 417 portfolios of CO230 Communications Skills V students. 

Each portfolio had two objective readers, and to ensure validity, no instructor was allowed to rate his or 
her students. The portfolios contained edited pieces of business communications — a cover letter and 
resume, a letter or memo — and printouts of six PowerPoint slides. CO230 faculty created the rating 
rubrics, based on course goals and practices. Faculty shared the rubrics with their students beforehand so 
that students knew how they were being assessed and that assessment was to be taken seriously. 





COPY AVAILABU! 



(Attached are the rating procedures, a list of items rated, and the rubrics of rating standards by which each 
portfolio was judged.) 



Analysis and Comments 

The( f CO230 faculty in a series of meetings between January and March 2002 defined the goals and items 
to be tested and then created appropriate rubrics (attached). One result of the meetings is the sense that all 
of the instructors understand the goals and criteria for assessment and are able to convey them to their 
students. This collaboration, an important social dimension of improvement, is bringing more coherence 
to the CSP by raising faculty awareness of program goals and helping them to focus on criteria in courses. 

On the day of the ratings, the faculty scored 417 portfolios, 18 of which were incomplete and could not be 
given a final rating. Of the ones rated, 105 were exemplary, 285 were proficient, and 9 did not meet 
standards. (See table below). Of the 462 students enrolled in CO230, 45 did not submit portfolios. 

Michael Schmoker (1996) in Results: The Key to Continuous School Improvement. (Alexandria, VA: 
Assoc of School and Curriculum Development) writes that the “combination of three concepts constitutes 
the foundation for results: 

• Meaningful teamwork 

• Clear, measurable goals, and 

• Regular collection and analysis of performance data (2).” 

The ongoing assessment of the Communications Skills Program contains all three elements, and is 
leading to useful results. In the Spring 2002 Term, student outcomes have been assessed through CO230 
portfolio assessment and objective testing (a follow-up test of all CO230 students who were originally 
tested in December 2000 with the same objective test). 

Conclusion 

The results suggest that the Communications Skills Program Courses I-V is achieving designated student 
outcomes basic PowerPoint design and production of cover letters, resumes, and memos. We should 
expect a high proficiency rate in CO230 since students have completed CO 1 20-22 1 . Some transfer 
students may be less proficient, but, in any event, they have to take C022 1 . CO230 students demonstrated 
that nearly 94 percent of them are exemplary or proficient in their abilities to write basic business 
communications that employers value. 

SUMMARY OF CO230 PORTFOLIO ASSESSMENT RESULTS, SPRING 2002 



Total Portfolios Scored 


417 






Total CO230 Enrollment 


462 














No Submissions 


45 


10% 


of 462 


Portfolios Scored 


417 


90% 


of 462 










SCORES (6 highest-1 lowest) 


Total 


% of 417 




Scores of 6 (Highest) 


6 


1.4% 




Exemplary (5.9-5) 


99 


23.7% 




Proficient (4-3) 


285 


68.3% 




Does Not Meet Standard (2-0) 


9 


2.1% 




Incomplete Submissions 


18 


4.3% 








100% 
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Contents of Portfolio 



1. COVER LETTER AND RESUME 

Assignment: The students will create a written resume and cover letter that follows the specified 
guidelines established from the text and the course presentation formats shared from Career Service. The 
student’s including the advertisement/description of job for which he or she is applying is optional. 
Evaluation: The cover letter and resume will be evaluated using a rubric that focuses on correctness of 
form and style, grammar and syntax, diction, etc. 

2. MEMO/LETTER "Why I Deserve to be Promoted: An Assessment of My Communication Skills" 
(must address writing, speaking, listening, and reading and research and critical thinking/problem solving 
skills. The student may submit in a letter or memo format that observes accepted business conventions. 
Assignment: The students will compose a metacognitive essay expressing specific strengths and 
weaknesses in the four communication strands {reading, writing , listening , and speaking). 

Evaluation: The essay will be evaluated in terms of content (student comments relative to specific 

strengths and weaknesses) and mechanics of effective writing using a specific rubric for writing. 

3. PPT SLIDES (5) from any assignment with printouts in handouts and in notes pages. 

Assignment: The students will create a basic PowerPoint 2000 presentation that includes minimally a 
title slide and slides that demonstrate a variety of PPT strategies such as bulleted lists, graphics (optional), 
and a pleasing layout. The slides should have no major grammatical or mechanical errors. An 
introductory slide is optional. 

Evaluation: The PPT presentation will be evaluated based on a rubric focusing on appropriateness of 
form and style, grammar and syntax, spelling, and other specified performance criteria as outlined in the 
Lehman, Creating Dynamic Multimedia Presentations text. 

Dates and Procedures 

Students turn in their final work for assessment by the last Friday of class (April 19) before final week. 
That Saturday, April 20, we have an all-day assessment (8 AM-5 PM); a continental breakfast and lunch 
will be provided. Each portfolio gets two readers excluding the instructor. Readers are volunteers and will 
be compensated. We will do holistic scoring, and keep what we rate simple because there are 23 sections 
of CO230 with 472 students. 

3 Ratings: Exemplary, Proficient, Does Not Meet Standard 



Description of Rating Procedures for the Day 

Twelve Raters scored 417 portfolios between 8:30 AM and 2:00 PM. 

Faculty were asked to be thoroughly conversant with the rubrics before they came to the session. 

At 8:00 AM, raters organized the folders into three groups. We assigned folders randomly, 
cautioning raters not to rate students from their own sections. 

At 8:30 AM rating began. First readers pulled portfolios at'random from their assigned alpha- 
piles, being careful not to rate students they’re currently teaching or whose work they might not be able to 
rate objectively. 
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Each reader assigned a holistic score (1-6) to each of the three components and an overall holistic 
score for the portfolio. 

The first reader wrote all four scores on the back of the folder. The folder is then placed in a 
“Second Readers” pile on the table. 

A second reader took portfolios at random from the “Second Readers” pile, being careful to avoid 
both portfolios for his/her current students and portfolios that he/she has already rated. The reader must 
check to see whether the holistic score for the portfolio (not the subscores) he/ she has assigned is roughly 
the same as that assigned by the first reader. If it is, the folder was “Completed” and given to Tom 
Marshall for recording the scores on an alpha list of all CO230 students. If it is not, the folder goes onto a 
“Resolutions” pile for a third reading. 

Since there few folders during the day in the “Resolutions” pile, no impromptu calibrations were 
needed to address different standards that raters were using to assess student materials. 

A third reader took portfolios from the “Resolutions” pile and re-scored, recording the score on 
the back of the folder. In fact, fewer than ten portfolios had to be rescored. 

At noon, there was an hour lunch break. All scoring was completed by 2:00 PM. 
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RUBRICS USED IN PORTFOLIO ASSESSMENT 
Cover Letter & Resume 



Resume and 
Cover Letter 


Exemplary (6-5 points) 


Proficient 
(4-3 points) 


Does Not Meet Standard 
(2-0 points) 


Rating 


Resume Heading 


Lists all necessary contact data: 
name, address, telephone number 


Lists all necessary 
contact data 


Omits key contact data 




Resume 
Objective, if 
listed 


Lists specific job objective with 
appropriate keywords 


Lists general objective 
appropriate for job 
sought 


Poor objective or 
inappropriate for targeted 
position 




Resume 

Education 


Lists all necessary data: Name of ■ 
degree, date of graduation, & 
institution. Lists major & GPA. 


Lists all necessary data 


Omits key data or listing is 
unclear 




Resume Work or 
Relevant 
Experience 


Itemizes jobs, most recent first. Lists 
most significant job title, employer 
and city, & dates of employment 
(month, year) 


Lists most necessary 
data 


Omits key data or data is 
out of proper order. i 




Resume 
Description of 
Work or 
Relevant 
Experience 


Uses appropriate action verbs and 
parallel structure to describe skills 
and summarize achievements, which 
are also quantified. Gives evidence of 
communication, management, and 
interpersonal talents. 


Uses appropriate action 
verbs and parallel 
structure; may not 
quantify all 

achievements; provides 
some evidence of 
talents 


Omits description; does 
not use action verbs or 
parallel structure; includes 
irrelevant data 




Resume Special 
Skills, 

Achievements, 
Awards, if 
applicable 


Highlights computer skills, offers 
evidence of being well-rounded 
individual by listing awards, 
experiences, and extracurricular 
activities. 


Lists most necessary 
data 


Poorly organized, includes 
irrelevant data such as 
personal information 




Audience of 
Resume and 
Cover Letter 


In workplace terms, it is ready to 
send out to external audiences or to 
higher ups "as is." 


With minor revisions, is 
ready for external 
audiences 


Exhibits little or no sense 
of audience. 




Organization 
and Layout of 
Resume and 
Cover Letter 


The layout is aesthetically pleasing 
and contributes to the overall 
message with appropriate use of 
headings and subheadings and white 
space. 


The layout uses 
horizontal and vertical 
white space 
appropriately. 


The layout is cluttered, 
confusing, and does 
not use spacing, headings 
and subheadings to 
enhance the readability. 




Conventions of 
Resume and 
Cover Letter 


Conforms to business letter and 
resume format. 


Conforms to business 
letter and resume 
format with minor 
revision. 


Does not conform to 
business letter and resume 
format 




Mechanics of 
Resume and 
Cover Letter 


The text is written with no errors in 
grammar, capitalization, punctuation, 
and spelling. 


The text is clearly 
written with little or no 
editing required for 
grammar, punctuation, 
and spelling. 


Errors in spelling, 
capitalization, punctuation, 
usage and grammar 
repeatedly.distract ihe 
reader and major editing 
and revision is required. 
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Memo/Letter Rubric 



Memo/ 

Letter 


Exemplary (6-5 points) 


Proficient 
(4-3 points) 


Does Not Meet Standard 
(2-0 points) 


Rating 


Introduction 


The introduction presents the 
main idea and draws the 
audience into the presentation 
with compelling questions or by 
relating to the audience’s 
interests or goals. 


The introduction is clear 
and coherent and relates 
to the topic. 


The introduction does not 
orient the audience to what 
will follow. 

The sequencing is unclear 
and does not appear 
interesting or relevant to the 
audience. 




Content 


The content is written clearly 
and concisely with a logical 
progression of ideas and 
supporting information. 


The content is written 
with a logical 
progression of ideas and 
supporting information. 


The content lacks a clear 
point of view and logical 
sequence of information. 

Includes little persuasive 
information and only one or 
two facts about the topic. 

Sequencing of ideas is 
unclear. 




Audience 


Appeals persuasively to values, 
capabilities, self-perceptions of 
primary audience. 


Frames message in ways 
that reflect likely values, 
capabilities, constraints 
of primary audience. 


Ignores audience values and 
constraints; shows no or 
little sense of audience. 




Layout and 

Business 

Format 


The layout is aesthetically 
pleasing and contributes to the 
overall message with 
appropriate use of headings and 
subheadings and white space. 

Format follows business/memo 
conventions 


The layout uses 
horizontal and vertical 
white space 
appropriately. 

Format follows 
business/memo 
conventions but needs 
minor revisions 


The layout is cluttered, 
confusing, and does not use 
spacing, headings and 
subheadings to enhance the 
readability. 

Format shows little or no 
awareness of business 
conventions 




Writing 

Mechanics 


The text is written with no 
errors in grammar, 
capitalization, punctuation, and 
spelling. 


The text is clearly 
written with little or no 
editing required for 
grammar, punctuation, 
and spelling. 


Errors in spelling, 
capitalization, punctuation, 
usage and grammar 
repeatedly distract the 
reader and major editing 
and revision is required. 

(more than 5 errors) 




Composite Rating: 6-5 (Exemplary) 4-3 (Proficient) 

2-1 (Does Not Meet Standard) 


TOTAL POINTS 
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PowerPoint Rubric (Part 1) 



PowerPoint 

Slides 


Exemplary (6-5 points) 


Proficient 
(4-3 points) 


Docs Not Meet 
Standard (2-0 points) 


Points 


Introduction 

(optional) 


The introduction presents the overall 
topic and draws the audience into the 
presentation with compelling questions 
or by relating to the audience's interests 
or goals. 


The introduction is 
clear and coherent 
and relates to the 
topic. 


The introduction does 
not orient the audience 
to what will follow. 

The sequencing is 
unclear and does not 
appear interesting or 
relevant to the audience. 




Content 


The content is written clearly and 
concisely with a logical progression of 
ideas and supporting information. 

The project includes motivating 
questions and advanced organizers that 
provide the audience with sense of the 
project’s main idea. 


The content is written 
with a logical 
progression of ideas 
and supporting 
information. 


The content lacks a clear 
point of view and logical 
sequence of information. 

Information is 
incomplete, out of date 
and/or incorrect. 
Sequencing of ideas is 
unclear. 




Text Elements 


The fonts are easy-to-read and point 
size varies appropriately for headings 
and text. 

Use of italics, bold, and indentations 
enhances readability. 

Text is appropriate in length for the 
target audience and to the point. 

The background and colors enhance the 
readability of text. 


Sometimes the fonts 
are easy-to-read, but 
in a few places the 
use of fonts, italics, 
bold, long 
paragraphs, color or 
busy background 
detracts and does not 
enhance readability. 


The text is extremely 
difficult to read with 
long blocks of text and 
small point size of fonts, 
inappropriate 
contrasting colors, poor 
use of headings, 
subheadings, 
indentations, or bold 
formatting. Slides are 
crowded (exceed 8x8+) 




Layout 


The layout is aesthetically pleasing and 
contributes to the overall message with 
appropriate use of headings and 
subheadings and white space. 


The layout uses 
horizontal and 
vertical white space 
appropriately. 


The layout is cluttered, 
confusing, and does 
not use spacing, 
headings and 
subheadings to enhance 
the readability. 
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PowerPoint Rubric (Part 2) 



PowerPoint 

Slides 


Exemplary (6-5 points) 


Proficient 
(4-3 points) 


Does Not Meet 
Standard (2-0 points) 


Points 


Graphics 

(optional) 


The graphics assist in presenting an 
overall theme and make visual 
connections that enhance understanding 
of concept, ideas and relationships. 

Original images are created using 
proper size and resolution, and all 
images enhance the content. 

There is a consistent visual theme. 


The graphics visually 
depict material and 
assist the audience in 
understanding the 
flow of information 
or content. 

Original images are 
used. 

Images are proper 
size, resolution. 


The graphics are 
unrelated to the content. 

Graphics do not enhance 
understanding of the 
content, or are 
distracting decorations 
that create a busy feeling 
and detract from the 
content. 




Writing 

Mechanics 


The text is written with no errors in 
grammar, capitalization, punctuation, 
and spelling. 


The text is clearly 
written with little or 
no editing required 
for grammar, 
punctuation, and 
spelling. 


Errors in spelling, 
capitalization, 
punctuation, usage and 
grammar repeatedly 
distract the reader and ’ 
major editing and 
revision is required, 
(more than 5 errors) 




Composite Rating: 6-5 (Exemplary) 4-3 (Proficient) 

2-1 (Does Not Meet Standard) 


TOTAL POINTS 
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